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Wednesday, June 22, 2011 


Peer review: Still no practical alternative? 


The UK House of Commons Science & Technology Select Committee is currently 
conducting an inquiry into peer review. It held its fourth oral evidence session on 
June ath, taking evidence from both funders of scientific research and from 
Government. 


UPDATE: THE COMMITTEE’S REPORT HAS NOW BEEN PUBLISHED. THE 
DETAILS ARE AVAILABLE HERE. 


While the Committee gave no specific reason for launching the current inquiry it 
seems evident from the questions MPs have been asking that two particular 

incidents have been exercising their minds: the long-running saga over Andrew 
Wakefield and the MMR vaccine scare, and the so-called Climategate incident. 


In the June 8'" session the Committee asked questions not just about the efficacy of 
peer review, but about scientific fraud, bias, the willingness of universities to 
investigate allegations of misconduct, and the need to make research data freely 
available so that others can access, examine and test them. 


MPs seemed particularly concerned that universities may be unwilling to investigate 
claims of misconduct. As MP Graham Stringer put it in one of the questions he asked 
the witnesses, “There is a certain amount of evidence that very little fraud is detected 
in universities and major research institutions in this country. Do you think we should 
be doing more to try and detect that, because in one sense there is an interest within 
those bodies not to discover or expose the problems they have, to sweep it under the 
carpet, isn’t there? If you are running a university and you find you have a researcher 
who just writes down his figures without doing the work, which has happened in one 
or two cases, the university doesn’t want to say that it has been employing a 
fraudster for 10 years, does it?” 


Politicians also probed the witnesses about the use of journal impact factors as a 
“proxy measure for research quality” when assessing the performance of academics, 
and whether “the growth of online repository journals” like PLoS ONE is a “technically 
sound” development. 


Robust defence 


For their part the witnesses put up a robust defence of current practices. They denied 
that universities would cover up fraud; they dismissed suggestions that the impact 
factor is used as a proxy measure of quality; and they insisted that, while it might not 
be perfect, there is no practical alternative to traditional peer review. 


In support of the latter claim they repeated the oft-made analogy with Winston 
Churchill’s description of democracy. Churchill famously described democracy as 
“the worst form of government, except for all those other forms that have been tried 
from time to time.” Thus it is with peer review, averred the witnesses: no one has 
come up with anything better. 


Those with any experience or knowledge of how peer review works in practice might 
have been tempted to conclude that analogising peer review with democracy is to 
obfuscate the issue. At the very least, it appears oxymoronic. 


Such a conclusion was all the more likely in light of the opening question and answer. 
The Chair suggested that it might be helpful to conduct some research into the 
efficacy of the current system — on the grounds that “evaluation of peer review is 
poor”. To this Wellcome Trust director Sir Mark Walport replied: “Peer review is no 
more and no less than review by experts. | am not sure that we would want to do a 
comparison of a review by experts with a review by ignoramuses.” 


Sir Mark’s statement can only have served to remind the audience that peer review is 
more oligarchic than democratic in effect. Rather than encouraging egalitarianism, it 
promotes elitism, and all the privileges one might associate with an old boy’s club 
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(appositely perhaps, there was not a single female witness called to give evidence on 
June ath). 


Of course the Churchillian analogy is not really meant to suggest that peer review is 
a democratic process. Nevertheless the witnesses’ repeated claims that the current 
peer review system is “good enough” would surely be challenged by many junior 
researchers, who frequently complain that scholarly journals tend to be controlled by 
small elite groups of insiders, invariably senior researchers. 


As one researcher pointed out to me recently, this is particularly problematic for 
those working outside North American and Europe. As he put it, “Peer-reviewed 
journals with a high impact factor are either dominated by certain gangs, or groups, 
or the editors rely on the opinion of reviewers too much.” 


The upshot, he added, is that “a small guy from Russia, Brazil or Thailand will never 
get published, even with excellent results, unless he or she has a prominent Western 
colleague as a co-author.” 


In fact, it is not just researchers in less privileged parts of the world who can struggle 
to get published in scientific journals today. Nor is it only junior researchers who 
complain about the peer review system. In 2009, for instance, 14 leading stem cell 
researchers wrote an open letter to journal editors highlighting their disquiet at the 
way in which the system operates. 


Speaking to the BBC about the letter Professor Lovell-Badge commented: "It's 
turning things into a clique where only papers that satisfy this select group of a few 
reviewers who think of themselves as very important people in the field is published.” 


Responding in a (separate) BBC interview, Sir Mark downplayed the criticism. 
Scientists, he said, “are always a bit paranoid” about peer review. And to make his 
point Sir Mark again used the analogy with democracy — peer review is not perfect, 
but it is the best system that the research community has been able to come up with. 


Risks 


As indicated, the June 8" witnesses also denied that the journal impact factor is 
used to evaluate researchers’ performance. “[W]e are very clear that we do not use 
our journal impact factors as a proxy measure for assessing quality,” David Sweeney, 
the director for research, innovation and skills at the Higher Education Funding 
Council for England (HEFCE) told MPs. “Our assessment panels are banned from so 
doing.” 


Once again, however, we could expect researchers to refute claims that journal 
impact factors are not used for promotion and tenure purposes. Whatever funders do 
or do not do, they might say, universities (that is, their employers) make it quite clear 
that being published in high impact journals is one of the best ways to advance an 
academic career. 


Indeed, some universities (perhaps not in the UK, but certainly in other countries) 
operate schemes in which researchers are given cash bonuses if they succeed in 
being published in a high-impact journal. Universities in China, The Netherlands and 
Egypt are said to provide such incentives. In the case of Cairo University the details 
are publicly available on the institution’s web site. The accompanying table shows 
that faculty members published in a prestigious journal can earn a lump sum 
payment of between 2,000 and 100,000 Egyptian pounds (£5,000 to £10,000), with 
the amount paid directly related to the impact factor of the journal. 


But it would be wrong to imply that the June 8'h witnesses were averse to new 
developments. They talked positively about the ways in which the Internet could 
improve the system, and they spoke enthusiastically about new online journals like 
PLoS ONE. “PLoS ONE has very good peer review’, said Sir Mark. They also cited 
with approval the development of web-based post-publication services like Faculty of 
1000. 


They were far less enthusiastic, however, about social networking tools like blogs, 
Twitter and Facebook. In contrast to PLoS ONE and Faculty of 1000 — which have 
been developed by, and are managed by, traditional publishers — the former tools 
were viewed as dangerously anarchic, uncontrolled, and uncontrollable. 


Describing the risks inherent in such services Sir Mark commented, “You have only 
got to look at the world of blogs, Twitter or anything else. Openness brings its own 
risks. If anyone can comment, then they can all say what they want, so of course 
there are risks like that.” 


Professor Rick Rylance, chair-elect for Research Councils UK appeared to agree: 
“You could end up in the rather ludicrous receding world of having to peer-review the 
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post-review and the rest of it to find out whether it has worth.” 


It is essential, added Rylance, for journals to continue to act as a quality filter 
“Clearly, if those filters are removed, there is a danger that people will be relatively 
unbuttoned about things.” 


Why this was problematic was not entirely clear, particularly as Sir Mark had earlier 
pointed out that in the humanities, “there is a long tradition of writing book reviews 
where one academic is scathingly rude about another academic.” Was the 
suggestion that, while it might be fine for humanities scholars, scientists should be 
preserved from scathing criticism? 


“| hope they are a threat” 


Not every witness evinced such a conservative view. Sweeney appeared to 
represent a more radical school. A little anarchy, he implied, might not be such a bad 
thing. “I think those risks exist but there are benefits,” he suggested. “We will have to 
adjust to the use of social networking in this area.” 


And when the witnesses were asked if article-level metrics posed a threat to high 
impact journals, Sweeney commented: “I don’t care if they are a threat to the base 
journals because the journal ecology will develop based on competition and 
alternative ways of doing things. | am sure they will respond. In some ways, | hope 
they are a threat.” 


Nonetheless, the overall impression created by the witnesses was that any change to 
the current system should be avoided, and that traditional peer review should be 
treated as sacrosanct — on the grounds that, whatever its drawbacks, it is the only 
practical way of ensuring the quality of published research. As indicated, there even 
appeared to be resistance to testing its efficacy. 


Again, many would disagree that published research achieves the levels of quality 
implied by the witnesses. They would also challenge the proposition that traditional 
peer review remains the only acceptable way of doing things. 


And they might add that, as a result of the intolerable pressure put on researchers to 
publish as many papers as possible, the quality of peer-reviewed research is actually 
falling. Finally, they might point out that since practically all papers are eventually 
published anyway (in one journal or another), quality is a relative concept in the world 
of scientific publishing today. 


It is for these reasons that there are growing calls for change. Moreover, many 
believe that, far from being a threat to the quality of published research, social 
networking offers a viable alternative. 


Richard Smith, the former editor of the prestigious journal BMJ, for instance, has 
reached the conclusion that the very notion of pre-publication peer review is flawed. 
The best way of assessing a paper, he suggests, is after publication, not before. 


As Smith puts it, “The problem with filtering before publishing, peer review, is that it is 
an ineffective, slow, expensive, biased, inefficient, anti-innovatory, and easily abused 
lottery: the important is just as likely to be filtered out as the unimportant. The sooner 
we can let the ‘real’ peer review of post-publication peer review get to work the 
better.” 


And today, he suggests, there is an effective way of doing this. “For journal peer 
review the alternative is to publish everything and then let the world decide what is 
important. This is possible because of the internet.” 


In January Smith developed his views further on the BMJ blog. In a post entitled 
“Twitter to replace peer review?” he suggested that bloggers and tweeters may be 
more effective at assessing research than the traditional peer review system. 


Smith related how the blogosphere had drawn attention to a major flaw in a paper 
published (and peer reviewed) by the high-impact journal Science. This claimed that 
scientists were now able to predict human longevity with 77% accuracy. “One week 
after the paper was published the authors acknowledged that they had made a 
technical error,” said Smith, “and shortly afterwards Science issued an “expression of 
concern,” meaning ignore this paper.” 


Revolution underway 


It is worth noting that the current inquiry into peer review is by no means the first 
such inquiry. It is further worth noting that all previous inquiries reached similar 
conclusions to those expressed by the June gth 
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In 1989, for instance, the UK Boden Report concluded there are, "no practical 
alternatives to peer review for the assessment of basic research.” It added, however, 
that this was no cause for complacency. "Rather the reverse. Peer review does have 
problems both in principle and in practice." 


The same sentiment was echoed in a 1995 Royal Society report on peer review. 


The current inquiry, however, is taking place in a web-enabled world. Given this, and 
the increased pressure on researchers to publish, the Committee is confronted with 
two new questions: 1) Have peer review practices deteriorated to the point where the 
status quo can no longer be justifiably supported and, 2) does the Internet, and the 
new social networking tools developed for it, offer a viable alternative to traditional 
peer review? Might these new tools indeed prove a superior way of reviewing, 
filtering and adjudicating on the quality and value of new research? 


The most likely outcome, of course, is that the Committee will once again conclude 
that traditional peer review remains the only practical solution. And in doing so, it 
may even cite Churchill's views on democracy. 


But whatever the Committee concludes, we should not doubt that a revolution in peer 
review is underway. Today the status quo is under attack from young Internet-savvy 
researchers keen to shake the establishment's tree. 


Smith cited one example. There was a similar incident last year, when another 
Science paper became the target of criticism in the blogosphere. The authors of this 
paper, which claimed to have demonstrated that arsenic-based life forms are 
possible, were accused of publishing flawed and inadequate research. 


The firestorm of criticism was sufficiently compelling, and vocal, that Science 
subsequently published eight papers criticising the original research, and the paper’s 
authors appear to be back-pedaling. They have also agreed to release the bacteria 
so that other groups can try and reproduce their results. 


Incidents like this are becoming commonplace, and some now believe that we are 
witnessing the beginning of the end for traditional peer review. Interestingly, the new 
approaches emerging on the Web appear to hold out the promise of providing a 
more democratic way of reviewing papers, one moreover that could prove both more 
efficient, and more cost-effective. 


In short, there may now by a practical alternative to traditional peer review. 


These developments are inevitably viewed by the establishment as threatening, not 
least because it means giving up some of the control it has traditionally enjoyed. But 
to resist these new forces would be to sit like King Canute ordering the tide to retreat. 


Some of the issues explored by the committee on June 8' are highlighted in 
the edited questions and answers below the video. The asterisked headings 
are mine. 


The Chair of the Science & Technology Committee is Andrew Miller, Labour MP for 
Ellesmere Port and Neston. Other politicians to pose the questions below were 
Graham Stringer, Labour MP for Blackley and Broughton, Roger Williams, Liberal 
Democrat MP for Brecon and Radnorshire, Gavin Barwell, Conservative MP for 
Croydon Central, Pamela Nash, Labour MP for Airdrie and Shotts, Stephen 
McPartland, Conservative MP for Stevenage, David Morris, Conservative MP for 
Morecambe and Lunesdale, Stephen Mosley, Conservative MP for the City of 
Chester, and Stephen Metcalfe, Conservative MP for South Basildon and East 
Thurrock. (A full list of Committee members is available here). 


Details of the first three oral sessions can be found here, here, and here. 


The June 8th hearing was split into two sessions. 
7HAHHT 
== FIRST SESSION == 


Those giving evidence in the first session were Professor Rick Rylance, Chair-elect, 
Research Councils UK, David Sweeney, Director for Research, Innovation and Skills, 
HEFCE, and Sir Mark Walport, Director, Wellcome Trust. 


* On whether research funders should fund research on the effectiveness of 
peer review... 


Q250 Chair: ... Evaluation of editorial peer review is poor. Should you, as funders of 
research, contribute towards a programme of research to, perhaps, justify the use of 
peer review in publication and find out how it could be optimised?... 
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Sir Mark Walport: It all depends what you mean by "research". It is quite important 
to have a very straightforward understanding of what peer review is. Peer review is 

no more and no less than review by experts. | am not sure that we would want to do 
a comparison of a review by experts with a review by ignoramuses. 


Q251 Chair: That’s not very nice, is it? 


Sir Mark Walport: Having said that, we do conduct studies of peer review. The 
Wellcome Trust published a paper in PLoS ONE a couple of years ago in which we 
took a cohort of papers that had been published. We post-publication peer-reviewed 
them and then we watched to see how they behaved against the peer review in 
bibliometrics. There was a pretty good correlation, although there were differences. 
Experiments of one sort or another are always going on, but the fundamental 
question of whether you should compare expert review with just randomly publishing 
stuff | don’t think is something that anyone would be very keen to do. It lacks 
equipoise. 


David Sweeney: Through our funding of JISC and through our funding of the 
Research Information Network, much work has been carried out in this area and we 
remain interested in further work being carried out where the objectives are clear. 


Professor Rylance: Yes. We, too, would be open to trying to think about how that 
might be researched. We have to bear in mind that peer review is not a single 
phenomenon. It is peer review in relation to publication, grant awards, REF and so 
on. Again, there are differences between the natural sciences, the social sciences 
and the humanities. You would have to define the task a bit more carefully. We do, 
from time to time, undertake research on, for example, the influence of bibliometrics 
and its relationship to peer review, so work is going on in that way. 


* On whether peer review limits the emergence of new ideas... 


Q252 Chair: The Wellcome Trust highlighted a common criticism of peer review by 
saying: "It can sometimes slow or limit the emergence of new ideas that challenge 
established norms in a field." Do the others agree and what can be done about this? 


Professor Rylance: Churchill once said that democracy was the worst system in the 
world apart from all the others. | think the same about peer review. Peer review is 
absolutely crucial, but, of course, it carries limitations of one kind or another in that it 
can slow down things. The volume of work load and so on and so forth is increasing 
but, none the less, we need to remain committed to the principle of doing peer review 
because, in the end, it is always the first and last resort of quality. 


David Sweeney: We think that there is a risk, but we also look at the many 
experiments that are going on with social networking and modern technological 
constructs. We hope that the broad view that is taken of those will mitigate the risks 
which the Trust identified. 


Sir Mark Walport: To be clear, the Wellcome Trust, in our submission, said: "Other 
commonly raised criticisms of peer review are..." We didn’t say that we agreed with 
that criticism. The issue is that peer review or expert review is as good as the people 
who do it. That is the key challenge. It has to be used wisely. It is about how the 
judgment of experts is used. It is about balancing one expert opinion against another. 
The challenge is not whether peer review is an essential aspect of scholarship 
because there is no alternative to having experts look at things and make 
judgments. 


* On whether the growth of “online repository journals” like PLoS ONE is 
“technically sound”... 


Q253 Chair: If that common criticism has validity, is the growth of online repository 
journals like PLoS ONE technically sound? 


Sir Mark Walport: It is entirely sound. PLoS ONE has very good peer review. 
Sometimes there is a confusion between open access publishing and peer review. 
Open access publishing uses peer review in exactly the same way as other journals. 
PLoS ONE is reviewed. They have a somewhat different set of criteria, so the PLoS 
ONE criteria are not, "Is this in the top 5% of research discoveries ever made?" but, 
"Is the work soundly done? Are the conclusions of the paper supported by the 
experimental evidence? Are the methods robust?" It is a well peer-reviewed journal 
but it does not limit its publication to those papers that are seen to be stunning 
advances in new knowledge. It is terribly important to put to bed the misconception 
that open access somehow does not use peer review. If it is done properly, it uses 
peer review very well. 


Professor Rylance: It is important to distinguish between peer review that is looking 
at a threshold standard, i.e. "Is this worthy of publication?" and peer review that is 


trying to say, "What are the best?" when you are over-subscribed in terms of the 
things you can publish. 


* On whether the impact factor of the journal a paper is published in should be 
used as a proxy measure of quality when evaluating researchers... 


Q255 Stephen Mosley: We have heard that the quality of journals, often determined 
by the impact factor of those journals, is becoming a proxy measure for research 
quality. Would you tend to agree with that assessment? 


David Sweeney: With regard to our assessment of research previously through the 
Research Assessment Exercise and the Research Excellence Framework, we are 
very clear that we do not use our journal impact factors as a proxy measure for 
assessing quality. Our assessment panels are banned from so doing. That is not a 
contentious issue at all. 


Sir Mark Walport: | would agree with that. Impact factors are a rather lazy surrogate. 
We all know that papers are published in the "very best" journals that are never cited 
by anyone ever again. Equally, papers are published in journals that are viewed as 
less prestigious, which have a very large impact. We would always argue that there 
is no substitute for reading the publication and finding out what it says, rather than 
either reading the title of the paper or the title of the journal. 


Professor Rylance: | would like to endorse both of those comments. | was the chair 
of an RAE panel in 2008. There is no absolute correlation between quality and place 
of publication in both directions. That is you cannot infer for a high-prestige journal 
that it is going to be good but, even worse, you cannot infer from a low-prestige one 
that it is going to be weak. Capturing that strength in hidden places is absolutely 
crucial. 


* On whether the 2014 Research Excellence Framework will use impact factor 
as a measure of quality ... 


Q256 Stephen Mosley: We have had some very good feedback about the RAE 
process in 2008 and the fact that assessors did read the papers, did understand 
them and were able to make a subjective decision based on that. But we have had 
concerns. | know that Dr Robert Parker from the Royal Society of Chemistry has 
expressed a concern that the Research Excellence Framework panels in the next 
assessment in 2014 might not operate in the same way. Can you reassure us that 
they will be looking at and reading each individual paper and will not just be relying 
on the impact? 


David Sweeney: | can assure you that they will not be relying on the impact. The 
panels are meeting now to develop their detailed criteria, but it is an underpinning 
element in the exercise that journal impact factors will not be used. | think we were 
very interested to see that in Australia, where they conceived an exercise that was 
heavily dependent on journal rankings, after carrying out the first exercise, they 
decided that alternative ways of assessing quality, other than journal rankings, were 
desirable in what is a very major change for them, which leaves them far more 
aligned with the way we do things in this country. 


Q257 Stephen Mosley: That is a fairly conclusive response, is it not? Lastly, you 
were talking about PLoS ONE in answering the Chair’s questions. From what you 
were saying, there is a difference in standard between papers in PLoS ONE that 
might not be in that 5% most excellent bracket, but just so long as the work is 
technically sound and correct, they are in there without being excellent. With the 
impact factor of those repository journals gradually increasing, does it mean that the 
proxy use of peer-reviewed publications is even a less valid approach to assessing 
the quality of research in institutions in the future? 


David Sweeney: | think we just don’t do that. We are not keen to do that. We want to 
assess-all the time we do work every few years-on how much we can use 
bibliometrics in a robust way, particularly as you aggregate the information over a 
large number of publications. At present we do not feel that the role that that should 
play is beyond informing the expert judgments that are made by panels. We are very 
conscious of the fact that our research assessment exercise has to go across all 
disciplines. There would be little argument that the use of metric information is really 
quite difficult in many disciplines. We are trying to have a consistent way of doing 
things. We are very keen to be abreast of the latest research but confident that peer 
review should remain the underpinning element. 


Sir Mark Walport: If you are assessing an individual, there is simply no substitute for 
looking at their best output. If you are assessing a field, that is when you can start 
using statistical measures. You can start using things like the number of citations. If 
you look at most funders, they are very focused on asking people to tell them what 
their best publications are, sometimes limiting the numbers. For our Investigator 
Awards, we limit the number of publications to people’s best 20. 


Professor Rylance: Following on from David’s point, in my field, in the humanities, 

the majority of publications are not in journals. They are in other forms like books or 
chapters in books and so on. There simply is not the bibliometric apparatus to derive 
sound conclusions for that reason. 


* On whether formal training in conducting peer review should be 
compulsory... 


Q258 Pamela Nash: Given the importance of peer review in both academic research 
and publishing, do you think that formal training in conducting peer review should 
become a compulsory part of gaining a PhD? 


Sir Mark Walport: Part of the training of a scientist is peer review. For example, 
journal clubs, which are an almost ubiquitous part of the training of scientists, bring 
people together to criticise a piece of published work. That is a training in peer 
review. Can more be done to train peer reviewers? Yes, | think it probably can. PhD 
courses increasingly have a significant generic element to them. It is reasonable that 
peer review should be part of that. People sometimes talk about the opportunity cost 
of peer review. Peer review is a form of continuous professional development. It 
forces people to read the scientific literature and it gives a privileged insight into work 
that is not yet published. Most laboratories would involve, if not their PhD students, 
their early post-docs in peer review work. 


Professor Rylance: | would echo and support that. It seems to me that research is a 
collective enterprise and that anyone who wishes to enter that field either as an 
academic or in some other capacity needs to understand that. So an engagement 
with the work of others of a judgmental or other kind is really quite important as part 
of that process. 


Q259 Pamela Nash: | am aware that the "Roberts funding" provided training for PhD 
students until recently. Would any of you have any ideas on who could be 
responsible for continuing that funding for that training? 


Sir Mark Walport: That funding is available. For example, the Wellcome Trust funds 
four-year PhD programmes, so we are providing funding for a longer period. The 
research councils can speak for themselves, but the four-year model of the PhD is 
becoming well established and that gives universities the opportunity to provide that 
transferable skills training. 


Q260 Pamela Nash: But should specific peer review training be recommended when 
that funding is given? 


Sir Mark Walport: We are not prescriptive in what universities teach. As | said, that 
would be a reasonable component of it. 


* On reducing the burden on referees... 


Q261 Pamela Nash: ... Both Research Councils UK and the Wellcome Trust 
mentioned in their contributions to this inquiry that it would be favourable to reduce 
the burden — the bulk of the work — on referees of the peer review process. What 
would each of you propose to help streamline that process and reduce the burden on 
referees? 


Professor Rylance: ... One thing you can do is demand manage. If the burden is 
increasing, and we recognise that it is just in terms of volume and the complication of 
frequency, if you start to reduce the number of applications, that work load starts to 
reduce and the quality of peer review goes up, presumably, how do you demand 
manage in that situation? You could do it in a draconian way. You could, for example, 
say, "The quota for this university is whatever it is", based on historic performance. 
You could do it developmentally working with universities to filter their own 
application processes, such that ones which are not going to go anywhere in any 
reasonable scheme are filtered out at an early stage, or you could go for what, in the 
jargon, is called "triage" processes when you receive them. So you do a relatively 
light-touch first stage application and then you reduce others. 


My personal view — there are differences of opinion about this — is that measures 
like quotas have quite significant downsides, of which probably the most significant is 
that they would discourage adventurous, speculative, blue skies applications 
because, naturally, if you have a quota, people tend to be conservative about what 
they are putting in in order to try and gain the best advantage... 


David Sweeney: For us it is a volume problem. Obviously, more research is being 
done and more findings are being produced. We think that the amount that needs to 
go through the full weight of the peer review system need not continue to increase. 
Indeed, we are seeing initiatives in that. As part of our assessment exercise, we 
require four pieces of work over seven years from academics. In most disciplines, 


they will publish much more than that, but they do not submit it to the exercise 
because we are interested in selectively looking at only the best work. We would 
want to encourage academics to disseminate much of their work in as low burden a 
way as possible, but submit the very best work for peer review both through journals 
and then, subsequently, to our system. That is the only way to control the cost of the 
publication system. We must look for variegated ways of disseminating and quality- 
assuring the results. 


Sir Mark Walport: The first thing is that the academic community is still highly 
supportive of the fact that peer review is an intrinsic part of the scholarly endeavour. 
To put some numbers on it, between 2006 and 2010 the Wellcome Trust made about 
90,000 requests for peer review. We got about 50% usable responses. The response 
rate was a bit higher but not every referee’s report added value. That is a pretty good 
response rate, and much of that was international. We used the global scientific 
community to help review and they do that very willingly. People who are in 
environments where they know they cannot themselves get a Wellcome Trust grant 
are, nevertheless, willing to referee for us... 


* On the withdrawal of funding for the UK Research Integrity Office (UKRIO)... 


Q264 David Morris: Professor Rylance, have Research Councils UK and 
Universities UK withdrawn funding from UKRIO, and, if so, why? 


Professor Rylance:... The original RIO was set up primarily with a remit for the 
biomedical sciences. It was set up on a fixed-term basis through a multi-agency 
system, which | am sure you are aware of, that included not just the funding councils, 
research councils and the Department of Health, but Wellcome were involved and 
other bodies. When that came to the end of its term, we had to make a decision 
about whether to continue. In other words, funding had stopped. It was not a 
question of withdrawing it. Do we continue that funding or do we not? There was a 
sense of two things. One is that it was really important to establish a body that had a 
remit and that that body should cover a broader range of disciplines than was the 
case with the original RIO. Secondly, we needed to disentangle various sorts of 
functions which were caught up within that original body. Could one be, for example, 
both a funder and an assurer of it, because you are clearly in quite a complicated 
relationship? Also, could you be both an assurer and an adviser, because, clearly, if 
you are giving advice which then turns out to be wrong, you would then be policing 
your own mistake at some level... 


... The general conclusion was that, in its current format at that stage, RIO was not 
going to meet the sorts of needs that | have just described. We continued its funding 
for a little and we are now thinking about different ways in which we can put together 
a collective agreement on it, largely through, probably, a concordat style 
arrangement. The key player in this, just to complete the story, will be Universities 
UK. The reason why Universities UK are key to this is because they are not funders 
themselves of research. 


Q265 David Morris: Are you saying that it is moving more towards the subscription 
funding model? Is this a necessary change? 


Professor Rylance: It will be a subscription model in the sense that it will involve a 
series of agencies that will participate in the funding of it... 


... There is a genuine sense among the bodies that | have just described that we 
need a cross-disciplinary organisation to provide assurance in tandem to link up the 
various sorts of assurance mechanisms that each funder has, to look at consistency 
and so on and so forth. That will be done, as | have described, through a concordat 
arrangement largely run through UUK, but that is as far as we have got at the 
moment. 


Sir Mark Walport: ... The Wellcome Trust was fully supportive of Research Councils 
UK on this matter. Research integrity is important. There is no argument and no 
debate about that. The question is where the responsibilities lie for ensuring that it 
happens. We believe very strongly that the responsibility for the integrity of 
researchers lies with the employers, so by and large that is the universities for 
university academics. It is clearly the research institutes for people employed by 
research institutes. That is why we support moving to a concordat between research 
funders and the employers whose researchers we fund that it is their responsibility, in 
the same way that health and safety is a responsibility that is delegated to 
employers. Frankly, we did not believe that UKRIO in the form that it was constituted 
was delivering what we needed. 


* On whether universities are doing enough to address scientific fraud... 
Q273 Graham Stringer: ... At our last evidence session we had the Pro-Vice- 


Chancellor responsible for research at Oxford here — | could give you the exact 
quote but | will not read it — who, basically, said that in his experience there had not 


been an occasion when they had had to investigate somebody for fiddling their 
results for fraudulent practices in research. On the other hand, we had another 
witness who told us that, if research institutions had not sacked at least one person, 
then they were not trying. Taking Oxford as an example, if you take your assertion 
that it should be the employers, that indicates that the employers are not carrying out 
that job. Certainly, in the case of Wakefield with the MMR scandal, the employers of 
Wakefield did nothing. | will now come to my question. Doesn't that mean to say that 
there has to be a huge change in employers’ practices if your view was to be 
maintained? 


Sir Mark Walport: Employers are responsible for the integrity of their employees in 
all sorts of aspects of life. They are responsible in business for making sure that they 
do not commit fraud and that the accounting is done well. | can’t possibly comment 
on whether individual universities are immune from the malpractice of their 
employees. | do not think it alters the fact that, as in health and safety, and all sorts of 
other aspects, such as the good behaviour of employers in respect of how they deal 
with students, this is an employer’s responsibility. Increasingly, universities are taking 
this very seriously. Of course, you can pick examples of where things go wrong. You 
can pick examples of where peer review hasn’t worked well. The Wakefield sad story 
is a very good example of that. That paper should never have been published. But 
that is not an argument against organisations doing it well. In a sense, the 
importance of the concordat will be that it sets out in extremely clear terms what the 
relationship is and what the roles and responsibilities of universities as employers are 
for the integrity of their employees. 


Q274 Chair: It is clear that the universities would have responsibilities, but, taking 
your two examples of health and safety or fraud in conducting their business, in both 
of those instances there is an external regulator with statutory powers. 


Sir Mark Walport: The question is what those statutory powers should be. 
Ultimately, it is clear that a scientist who has committed some form of scientific fraud, 
if | can put it that way, should lose their job. Does that then fall under some other 
regulator? Is it something that the courts should deal with? Probably not very often. 
In the case of medical research, Andrew Wakefield eventually met his come-uppance 
at the General Medical Council. There are ways of doing this. 


Q275 Graham Stringer: But he did not, did he? He was struck off for bad ethical 
practice. The General Medical Council did not deal with whether his research was 
fraudulent or not. In a sense that is a bad example. If | can repeat Andrew’s point, 
yes, it is the employers’ responsibility, but who is going to keep the employers good? 


Sir Mark Walport: That is where the funders will play a very serious role. We take 
research integrity very seriously as well. It is a grant condition that the work is done 
properly. From our perspective, in relation to an institution that failed to manage the 
research integrity properly, we would have to question whether that was an institution 
at which we could fund research. It is not that we don’t take it seriously, but we 
believe that the mechanism for dealing with this has to be through the employer. 
Frankly, if the employer is unaware of things going wrong in the research, it is difficult 
to see how others would be aware and the employer would be completely unaware. 
They are doing it in whistleblowing procedures... 


Q276 Pamela Nash: If | could take up that point, without an external regulator — you 
have just said that funders have a responsibility here on who they fund — surely, that 
is then an incentive for an academic institution to keep things quiet so that they don’t 
lose funding. 


Sir Mark Walport: Not at all. It is the nature particularly of scientific research that 
errors are found out, and it can’t be in the interests of any good university not to have 
the research done to the highest possible standard....There is no incentive to cover 


up. 


* On whether there is a need to provide greater openness and transparency in 
scientific data ... 


Q277 Graham Stringer: ... Can! ... quote from last week’s Scientific American, 
which makes the point really well? ... It is by John PA. loannidis: "The best way to 
ensure that test results are verified would be for scientists to register their detailed 
experimental protocols before starting their research and disclose full results and 
data when the research is done. At the moment, results are often selectively 
reported, emphasising the most exciting among them, and outsiders frequently do 
not have access to what they need to replicate studies. Journals and funding 
agencies should strongly encourage full public availability of all data and analytical 
methods for each published paper." Do you agree with that and do you follow those 
policies? 


Sir Mark Walport: This is one of the arguments in favour of good peer review, 
because a good peer reviewer when reviewing a scientific paper actually probes and 
says, "Where are the controls? Where is the missing data?" That is the first thing. 
Secondly, we do explicitly ask investigators when they are generating datasets how 
they will handle the data. In general terms, we do encourage openness. In fact, at the 
moment there is a Royal Society inquiry on openness in science which is looking at 
the whole issue of openness of data. One has to recognise that there are both real 
costs and opportunity costs. Data is not an unalloyed good, as it were. It is 
something that has to be interpretable. It is quite easy to bamboozle by just putting 
out billions of numbers. It is actually a question of presenting the data in a way that is 
usable by others. But the principles of openness in science, of making data available 
and open, are something that the Wellcome Trust and other funders of biomedical 
research around the world are fully behind and completely supportive of. 


Q278 Graham Stringer: Is what lies underneath that answer that you believe that 
codes, computer programs and all the data that would enable other researchers to 
replicate the work should be made available publicly? 


Sir Mark Walport: Bearing in mind the feasibility and garbage in/garbage out, one 
has to be careful that the data is usable. Yes, increasingly very large datasets are 
generated. We want to maximise the value of the research that we fund. Therefore, 
openness is a very important principle. There are some other issues that need to be 
dealt with as well, so if you are dealing with clinical material then the confidentiality of 
participants is paramount. You have to manage data so that they are appropriately 
anonymised and people cannot be revealed. It has to be in the general interest of the 
advancement of science and knowledge. As you say, science is validated by its 
reproducibility. If you cannot see the data, that is a problem. Of course, the revolution 
of the power of the internet to make data available has meant that it is possible to put 
out data in ways that were never possible before. 


...Broadly, it makes complete sense to make as much data available in as usable a 
form as possible. That is something that we strongly support. It is why the funding of 
institutions like the European Bioinformatics Institute, which is housed at Hinxton, is 
so important. The UK Government has a good track record in supporting the EBI and 
funding has recently been announced for an extension there as part of the European 
ELIXIR project. Making data available is something that is incredibly important. 


David Sweeney: We believe in openness and efficiency in publicly funded research. 
Dr Malcolm Read took you through some of the issues at a previous hearing. We 
have funded and continue to fund projects that will push this area forward — UKRDS 
— and now some projects are looking at how cloud computing can help. Of course, 
we have learnt a lot from the research councils that the ESRC data archive has been 
a stunning success over many years... Technology is now allowing us to make 
advances, and through the work we fund we will learn a lot. Our objective is 
openness. 


Q279 Graham Stringer: Where research is publicly funded, if | can paraphrase what 
you Say, you are saying that the data should be publicly available. If there are good 
reasons for it being confidential, do you think it should be made available in a 
confidential depository to the reviewers and, potentially, for other researchers so that 
it is available in some form? 


David Sweeney: That requires consideration of the particular circumstances and the 
sensitivity. Reviewers should have access to all the information. They need to assure 
themselves of the quality. 


Professor Rylance: You start from that principle and then you think why it is that you 
shouldn’t reveal that rather than thinking you should close it and then why you should 
reveal it. 


* On the costs of storing large amounts of data... 


Q280 Graham Stringer: You have mentioned that you could have a huge dataset. 
Some of it may be good data and some of it may be rubbish. Are there real problems 
of costs and, if there are, who should pay for those costs of storage? Are there any 
other practical problems of storing huge datasets? 


Sir Mark Walport: There are very major costs. For example, the Sanger Institute this 
year alone has generated 1,000 human genome sequences. That is a massive data 
burden. Indeed, the costs of storing the data may in the future exceed the costs of 
generating it. Who should be responsible for doing that? It is, ultimately, a research 
funder issue, because we fund the research and so we have to help with the storage. 
It is like all of these things. Our funding is a partnership between the charity sector 
and the Government and it is a shared expenditure. 


Professor Rylance: There are issues as well about obsolescence. At what point 
does this data become simply not relevant anymore? The length of time for that will 
be discipline- specific and so on. There are a whole host of practical issues about 
how you do this. IP — intellectual property — is one, particularly, in my area, to do 
with creative works, for instance. 


* On the degree to which article level metrics can measure the quality of 
research, and whether they pose a threat to high impact journals... 


Q281 Stephen Metcalfe: | would like to turn now to the importance of articles versus 
journals, if | may. As | know you are aware, PLoS ONE instituted a programme of 
article level metrics. Do you believe that that is a good way to judge a piece of 
published science and, therefore, you are judging it on its intrinsic merit rather than 
the basis of the publication that it is in? 


Professor Rylance: Yes, absolutely. To echo what we were saying earlier on, it is 
intrinsic merit that we are after. It is not reputational or associational value. 


David Sweeney: | am not entirely sure that | would say that article level metrics 
necessarily captured the intrinsic metric merit. We should look at metrics of all kinds 
and try and judge where the collection and development of the metric does add 
value. As you drill down to individual articles, some metrics really are not entirely 
helpful. We have seen that with certain solid evidence in bibliometrics. Equally, we 
can see, with some of the networking metrics, that they may provide helpful 
information. | remain of the view that there will be no magic number or even a set of 
numbers that does capture intrinsic merit, but one’s judgment about the quality of the 
work, which may well be, in any way, in the eye of the beholder, may be informed by 
a range of metrics. 


Sir Mark Walport: | complete agree with David Sweeney on that. You can alter the 
number of times that an article is downloaded by merely putting some words in the 
title. There is good evidence that the content of the title influences the number of 
times that something is downloaded, so measuring download metrics can be very 
misleading. Different fields have different types of usage. Methods papers, typically, 
are extraordinarily heavily cited. There can be a long time before the importance of a 
paper is picked up. It is like all of these things; at a mass scale the statistics are 
helpful. If you want to assess the value of an individual article, | am afraid that there 
is no substitute for holding it in front of your eyes and reading it. 


Q282 Stephen Metcalfe: You don't see the article level metrics as a potential threat 
to the more established high impact journals. 


Sir Mark Walport: They are not a threat. Web-based publishing brings new 
opportunities, because it brings the opportunity for post-publication peer review and 
for bloggers to comment. There are things like the Faculty of 1000, which provides 
commentaries on papers. There are more and more ways for finding papers among a 
long tail of publications. This is a fast-evolving space. As the new generation of 
scientists comes through who are more familiar with social networking tools, it is 
likely that Twitter may find more valuable uses in terms of, "Gosh, isn’t this an 
interesting article?" All sorts of things are happening. It is quite difficult to predict the 
future. It can only be an enhancement to have the opportunity for post-publication 
peer review. It has turned out to be quite disappointing in that scientists have been 
surprisingly unwilling to put detailed comments. When the Public Library of Science 
started, it had plenty of space where you could comment. Academics are remarkably 
loath to write critical comments of each other alongside the articles. 


Q283 Stephen Metcalfe: Does anyone else want to add to that? ... 


Professor Rylance: No. |, personally, do not think it is a threat. There are two issues 
here. One is the recognition of merit. | entirely agree with my colleagues that, in the 
end, you have got to read the bloomin’ thing to see whether that is true. Then there is 
the issue about how people gain access to the good and the strong. That is a slightly 
different question. 


David Sweeney: | don’t care if they are a threat to the base journals because the 
journal ecology will develop based on competition and alternative ways of doing 
things. | am sure they will respond. In some ways, | hope they are a threat. 


Q284 Stephen Metcalfe: You touched upon scientists being unwilling to get heavily 
involved in post-publication peer review. Philip Campbell from Nature told us that that 
may well be — | am summarising here — because there is no prestige or credit 
attached to that particular role and there is the risk of alienating colleagues by public 
criticism. Do you agree with that? Do you think that there should be a system of 
crediting people? 


Sir Mark Walport: There are two separate issues. There are some very interesting 
community issues here. In the humanities, there is a long tradition of writing book 


reviews where one academic is scathingly rude about another academic. 


... In the case of the scientific world, that tearing apart is done at conferences and at 
journal clubs. The scientific community does not have a culture of writing nasty things 
about each other. This is an evolving world. 


Q285 Stephen Metcalfe: So introducing a system of credit- 


Sir Mark Walport: On credit, | think one has to be realistic. Are you going to promote 
someone on the basis of the fact that they wrote a series of comments on other 
scientific articles? The hard reality is that the core activities of an academic in terms 
of their promotion and pay recognition are going to be around their own scholarship 
and their own educational activities. It can only be at the margins that you will get 
brownie points for having done post-publication peer review. 


Q286 Stephen Metcalfe: Finally, if post-publication commentary were to grow, are 
you concerned about how you could ensure that there was no bias in that 
commentary, either positive or negative, either those wanting to build up someone’s 
reputation or those wanting to tear it down without anyone actually challenging 
them? 


Sir Mark Walport: It is quite clearly a risk. We see that in every other walk of activity 
on the internet. You have only got to look at the world of blogs, Twitter or anything 
else. Openness brings its own risks. If anyone can comment, then they can all say 
what they want, so of course there are risks like that. 


Professor Rylance: You could end up in the rather ludicrous receding world of 
having to peer-review the post-review and the rest of it to find out whether it has 
worth. Sir Mark was talking about the way humanities review each other’s things in 
print. Of course, one function for the journals that do that is to act as a quality filter to 
make sure that nothing defamatory, inaccurate or prejudiced is being said. Clearly, if 
those filters are removed, there is a danger that people will be relatively unbuttoned 
about things. 


Sir Mark Walport: It is self-correcting in that the scientific community is constantly 
scrutinising each other. A scientist who wrote something that was particularly 
egregious would be subject to the peer review of their own community. 


David Sweeney: | think those risks exist but there are benefits. We will have to 
adjust to the use of social networking in this area. 
HH 


== SECOND SESSION == 


Those giving evidence in the second session were Professor Sir John Beddington, 
Government Chief Scientific Adviser, and Sir Adrian Smith, Director General, 
Knowledge and Innovation, Department for Business, Innovation and Skills. 


* On whether the peer-reviewed literature is fundamental to the formation of 
Government policy... 


Q287 Chair: ... You are familiar with the piece of work that we are undertaking. We 
have heard that researchers perceive peer review to be "fundamental to scholarly 
communications". Is peer-reviewed literature also fundamental to the formation of 
Government policy? 


Professor Beddington: ... The answer to that question is that science and evidence 
is clearly fundamental to Government policy and peer review is a fundamental part of 
science evidence. That is not meant to be a cute response, but it is absolutely clear 
that the process of science involves peer review, and properly so, and that scientific 
evidence is essential for being the evidence-based policy of the Government. 


* On whether there is a need for research into the efficacy of peer review... 


Q290 Chair: Evaluation of editorial peer review is poor. Do you think that there is a 
need for a programme of research in this area to test the evidence for justifying the 
use and optimisation of peer review in evaluating science? 


Sir Adrian Smith: The short answer is no. It is an essential part of the scientific 
process, the scientific sociology and scientific organisation that scientists judge each 
other’s work. It is the way that science works. You produce ideas and you get them 
challenged by those who are capable of challenging them. You modify them and you 
go round in those kinds of circles. | don’t see how you could step outside of the 
community itself and its expertise to do anything other. You have probably had it 
quoted to you already, but there was a paper in Nature in October 2010 when six 


Nobel Prize winners were asked to comment on how they saw the peer review 
process. Basically, it was the old Churchillian thing that there are all sorts of 
problems with it but it is absolutely the best thing we have. 


Professor Beddington: Peter Agre makes that point in that same article, saying: "I 
think that scientific peer review, like democracy, is a very poor system but better than 
all others." 


Q291 Chair: That is twice that that has come up today. 


* On the benefits of codifying the use of peer review, and whether UK scientific 
advisory groups are mandated to use peer review... 


Q292 Stephen McPartland: | would like to ask you about Government use of peer 
review research. The US Congress has codified the use of peer review in 
Government regulations using the "Daubert Standard". In the US, the Supreme Court 
codified their use in the courtroom. Have you had any discussions with your 
American counterparts regarding how this works and what any of the benefits are? 


Professor Beddington: ... We would not see particular merit in excluding non-peer- 
reviewed information, because we have to recognise that there is a whole set of 
information that comes in as Government makes policy, some of it via the media, for 
example, evidence that is coming in to deal with emergencies. A basic decision on 
that | don’t think would be helpful. The issue is obviously going to be that, when we 
provide scientific advice to Government, there will be a weighing of that advice and 
the fact that certain advice is peer-reviewed and appropriately so, or indeed has 
been highly cited in a praiseworthy way, will go into the balance of that advice. | think 
| would advise against a piece of legislation saying that only peer review would be 
done. One would also have to question the definition of peer review and so on. | 
don’t think it would be something that | would be recommending to Government to 
think about adopting... 


Q294 Stephen McPartland: Do you believe that a test should be developed to 
identify whether or not peer review is reliable? This Committee recommended in 
2005, in a report entitled Forensic Science On Trial, that a test for expert evidence 
should be developed, building on the US Daubert test, and the Law Commission has 
now built on that and published a draft Criminal Evidence (Experts) Bill. 


Professor Beddington: | would think that this has to be thought about on a case-by- 
case basis. Peer review is not a homogeneous activity. If one is starting to see that 
there are, for example, problems of peer review in a particular journal or ina 
particular area of science, that needs to be addressed by that journal and by the 
people who work in that particular area of science. If you posed the question, "Is the 
peer review process fundamentally flawed?" | would say absolutely not. If you asked, 
"Are there flaws in the peer review process which can be appropriately drawn to the 
attention of the community?" the answer is yes. From time to time that will happen 
and that’s the way to do it. 


Sir Adrian Smith: And there will, from time to time, be misjudgments in that system. 
You can distinguish the system from particular cases within the system. 


Q295 Stephen McPartland: Are UK scientific advisory groups mandated to use peer 
review? 


Professor Beddington: No, for the very reasons | gave in my answer to the Chair’s 
earlier questions. We would certainly always take into account peer-reviewed 
information in providing advice to Government. | don’t think we would ever exclude it, 
but that would not be the sole evidence. In fact, some of the evidence that would 
come in would depend on the area of science. For example, in a large part of social 
science the scholarship is developed by the production of books, quite often well 
after the event. Yet social research is extremely important to Government policy. We 
would have this but it would not necessarily have been published in a social research 
journal. By contrast, for example, if we are thinking in the context of some work on 
genomics, then one would be expecting that to have been peer-reviewed and that 
would be going into the evidence. Again, | just don’t think that one would seek to 
make regulation. | emphasise again that the evidence we use in scientific, including 
social research, evidence, will sometimes be peer reviewed. Obviously, we would not 
seek to exclude peer-reviewed material but we would not wish to exclude material 
that had not been peer reviewed for these sorts of reasons. 


* On the effectiveness of peer review in validating assertions made in articles 
submitted for publication... 


Q296 Roger Williams: ... In your opinion how well does the peer review process 
validate the assertions made in articles put forward for publication? 


Professor Beddington: In a sense, both Adrian and | have answered that question 


earlier. Peer review does not guarantee that the results are correct. Science moves 
on by its use of scepticism and under challenge. We see all the time in the journals 
that are published this week that there will be people who have challenged peer- 
reviewed papers that were published some years ago and pointed out fundamental 
flaws in them or new evidence that undermines the conclusions of those papers. 
That is the progress of science. We can’t say that it is a guarantee, and manifestly 
not. 


We can say that it is an awful lot better than bare assertion without evidence. 
Particularly when you are looking at scientific issues that are fundamental to policy — 
| have talked about this to this Committee before — the emergence of scientific 
consensus is very important. That is not to say you do not have sceptics or 
appropriate challenges, but peer review does not guarantee that and it never 

could... 


* On whether peer reviewers should assess the underlying data supporting a 
research article as well as the article itself, and whether the raw data should be 
made freely available... 


Q297 Roger Williams: Today, and increasingly, | guess, in the future, submissions in 
science will be accompanied by very large and complex sets of data. Do you think 
that the reviewers should be assessing that underlying data as well as the article that 
is being produced? 


Sir Adrian Smith: In an ideal world, but that is rather difficult, is it not, because data 
will come out of laboratories and field studies. As a reviewer, you can’t go off and 
replicate that. If you are trying to study somebody’s derivation of a mathematical 
formula, you can replicate. The difference between the scientific argument and the 
data is rather different, but the protocols that are in place for collecting data, for 
example, in medicine, in conducting proper clinical trials and all the rest of it, are in 
an environment where all the pressures and checks and balances are to get that 
right. 


Q299 Roger Williams: Sir John, the Government is, obviously, a very substantial 
funder of science. Should it, as a matter of principle, require that all this raw data 
should be made available? 


Professor Beddington: Adrian has made a parallel point. With Government-funded 
science, the push is to have data out into the open. There are some areas, for 
example, shared data, which means you have a mix of data where some of the 
ownership of that data is outside the UK. You cannot make a hard and fast rule. In 
principle, though, the answer is that the more people who will look at the scientific 
problems from which we are wanting to get evidence the better. Therefore, 
transparency is, obviously, extremely attractive. From time to time, there will be 
timing issues, IP issues and so on, which will mean that transparency can be 
problematic. In the area we were looking at-the community of chief scientific advisers 
deals with this a lot of the time-we would be looking at material, and if it was not out 
in the open they would ask why not. If there is no good reason, they would urge that 
it would be put out into the open. Indeed, research councils push exactly along these 
lines. 


Sir Adrian Smith: There will always be issues of personal data protection, 
commercial interests and intellectual property and national security, so the situation 
is quite complex. | understand that the Royal Society will be doing a study sometime 
over the next 12 months that the Committee may well be interested in. 


Q300 Roger Williams: | think there is agreement that this data should be made 
available, subject to all the concerns that you have expressed about IP and 
commercial interests. Another matter is the cost of all this. Who should bear that cost 
if it is going to happen on a greater scale than it has in the past? 


Sir Adrian Smith: That is one of the issues that the Royal Society may well look at. 
Different communities, different cultures and different forms of data pose different 
issues, but there is a real problem... 


* On whether there should be a legal requirement on institutions to conduct a 
timely inquiry in cases of publication fraud or misconduct, and then publish 
details of the incident and the disciplinary action that has been taken... 


Q310 Gavin Barwell: In the past there has been a perception that publication fraud 
or misconduct has not always been investigated by the institutions in a timely 
fashion. Wakefield and MMR is an example. Should there be a legal requirement on 
institutions to conduct a timely inquiry and to publish the full findings of that inquiry 
and any disciplinary action that is taken? 


Sir Adrian Smith: | don’t know whether you need to go to what "legal" means, but, if 
you think of the funding that goes into universities, some of it will come through the 
Funding Council, for instance, through the QR stream and some through research 
grants. Both with the research councils and the Higher Education Funding Council 
conditions of grant are attached which make it clear what the expectations of 
behaviour are. | don’t think those are sufficient sanctions in themselves. An institution 
that would not follow up properly would be putting at risk its funding from HEFCE and 
the research councils. 


Q311 Gavin Barwell: Are there specific conditions relating to what institutions should 
do if there is a suggestion that misconduct was taking place? 


Sir Adrian Smith: Probably not. 
Q312 Gavin Barwell: Do you think there ought to be? 


Sir Adrian Smith: ...My own view, having run a university for 10 years, is that the 
constraints you are under in terms of conditions from the many funders that one has 
are quite sufficient to frighten one into doing appropriate things. 


Professor Beddington: The RCUK’s code of conduct, too, is a good look guideline 
in terms of conflicts of interest and appropriate behaviour. In the sense that 
universities depend on a significant income from the research councils, then they 
would be extremely unwise not to take forward any issues very quickly where they 
had detected fraud. The media would be commenting on it and other people in the 
same scientific area would be commenting on it. There would be a very substantial 
incentive for the universities to take this forward rather quickly. 


Q313 Graham Stringer: ... There is a certain amount of evidence that very little 
fraud is detected in universities and major research institutions in this country. Do 
you think we should be doing more to try and detect that, because in one sense there 
is an interest within those bodies not to discover or expose the problems they have, 
to sweep it under the carpet, isn’t there? If you are running a university and you find 
you have a researcher who just writes down his figures without doing the work, which 
has happened in one or two cases, the university doesn’t want to say that it has been 
employing a fraudster for 10 years, does it? 


Sir Adrian Smith: | would disagree. When | ran a university, | would put it exactly the 
other way round. The institutional reputation will suffer much more long-term harm if 
you allowed fraudsters to exist and you don’t do anything about it. In fact, | think you 
would get a lot of brownie points in many communities if you publicly identified such 
people and threw them out. | think the incentives are all in the opposite direction. 


Q314 Graham Stringer: It is surprising, therefore, is it not, ... that there are no 
cases in Oxford, as the Pro Vice Chancellor told us, and that there are very few 
cases in other universities and research institutes where people have found 
fraudulent behaviour? In the case of Wakefield, even when fraudulent behaviour was 
found out, the institution investigated itself and found nothing wrong. The evidence 
we have is in the other direction, isn’t it? 


Professor Beddington: | would not seek to comment on the Wakefield case. The 
issues here are that there is so much in the checks and balances in the way that 
science operates that fraudulent behaviour is highly likely to be detected by, initially, | 
suspect, gossip and then increasing concern that there is something wrong. That will 
happen. It may happen in the community and the attention will then be drawn to the 
university, and it would be very unwise for the university to ignore that information. | 
have not experienced it in 25 years at Imperial College. 


Q315 Graham Stringer: Can | ask why you won’t comment on Wakefield, because it 
is one of the great scandals of the last 10 or 12 years? It was not dealt with very well. 
Are there not things to be learnt from that? 


Professor Beddington: Yes, there are. My reason for not commenting is that | 
haven't read into it for a while, and | would like to re-familiarise myself before | 
commented, Mr Stringer, rather than any shyness on my part. | am not on top of the 
detail. 


* On whether researchers are biased towards the products of the 
pharmaceutical companies that sponsor their research... 


Q316 Graham Stringer: ... Are there problems with peer review in other areas? For 
instance, there is a huge amount of research sponsored by pharmaceutical 
companies and companies that produce biomedical products. Do you believe that a 
lot of researchers in those areas are biased towards the products that those 
companies are selling? 


Sir Adrian Smith: ... | don’t think a lot of the research itself is biased. There are 


biased reporting effects, because if you are doing clinical trials and you get negative 
results, there isn’t a journal on clinical trials that didn’t work. It is the ones that work 
that get published. There is a selection bias in that sense. Do not forget that at the 
end of the day these things have to get through the FDA or drug regulatory 
authorities if they are to come on to the market. Then you have incredibly close 
scrutiny of the protocols, the trials that were done, the conditions under which they 
were done and so on and so forth. | think there are tremendous checks and balances 
in the system against that. 


* On whether there is a problem with colleagues reviewing each other’s papers 
in those areas where only a small number of researchers are working... 


Q317 Graham Stringer: Are there structural problems where there are only three 
experts in a particular field, so that they are, effectively, all peer reviewing each other 
and they either agree or disagree? In one sense, that was the major criticism of 
those people who criticised the researchers at the university of East Anglia for their 
research, was it not? There is a very small pool of researchers in that area. 


Professor Beddington: Yes, you have that, but people are always moving out of 
their own fields. There is academic interchange. If things are of sufficient importance, 
they are likely to get challenged, not necessarily by the top two experts in the field 
but by others who are around the fringes, particularly if they are of significant 
interest... 


Q318 Graham Stringer: To finish on a fairly obvious question, nearly all of our 
witnesses have used the Churchillian quote, but when you get fraudulent papers that 
have been through the best process we have of peer review, do you think that peer 
review has damaged that process? Getting back to Wakefield, his paper was peer 
reviewed. Do you think the peer review process has been damaged? 

Sir Adrian Smith: How far do you want to take the Churchillian democracy analogy? 
There are bad things that happen within the peer review system. Not every MP who 
has been elected has behaved totally honourably. 

Graham Stringer: What a shocking thing to say. 

Sir Adrian Smith: You would not abandon the democratic process, presumably. 


Graham Stringer: No. That would be terrible. Thank you. 


Q319 Chair: Finally, are you aware that RCUK has ever cut funding because of fraud 
or allegations of fraud? If so, could you give us any examples? 


Sir Adrian Smith: | would have to go back and look through the archives, as it were, 
and directly ask that of chief executives. | am not directly aware of a case. 


Professor Beddington: | have no experience of it. 
The (uncorrected) transcript of the full session is available here. 


The video of the event can be accessed here, if it does not open up 
automatically in this post. 
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